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Safety Envelope Approach to ML Deployment joe 


m Specify unsafe regions 


=m Specify safe regions 
e Under-approximate to simplify 


= Trigger system safety response 
upon transition to unsafe region 


= Inherent tension of envelope 
simplicity vs. permissiveness 
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Architecting A Safety Envelope System Tay 
= “Doer’ subsystem Doer/Checker Pair 






e Implements normal, untrusted functionality Low SIL 
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jb 
=) 
Z 
= ‘Checker’ subsystem — Traditional SW = owen <° OUTPUTS 
e Implements failsafes (safety functions) SE ipl 
Za. 
a High SIL 
= Checker entirely responsible for safety & ae 
e Doer can be at low Safety Integrity Level Envelope 
e Checker must be at higher SIL Checker 


(Also known as a “safety bag” approach 
or monitor/actuator pair) © 2021 Philip Koopman 52 
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Self-driving shuttle company 
ordered to stop carrying 
passengers after injury 


The DOT suspends a shuttle operator on the same day it was 
criticized for being too hands-off 
By Sean O'Kane | @sokane1 | Feb 26, 2020, 12:56pm EST 
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-Self-driving shuttle company 
_adds seatbelts in order to 
resume US operations 


' But passenger rides might be scarce during the pandemic 


By Sean O'Kane | @sokane1 | May 18, 2020, 4:31pm EDT 
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8 SELF-DRIVING SHUTTLE BUS IN 


SPAIN’S MADRID PROVOKES CRASH 
ON FIRST DAY 


_ ia 
-y En el campus de Cantoblanco 


| ESTRENAN EL AUTOBUS AUTONOMO 
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The vehicle was travelling at a speed of 20 kilometres per hour through the Universidad 
Autonoma de Madrid when it provoked the accident. 


By Cristina Hodgson - 23 Oct, 2020 @ 10:00 m0 


https://bit.ly/3udw3ie 
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Physics-Based Checker Rules TS i, 


= Responsibility-Sensitive Safety (RSS) : 
e Safe distances based on physics 
e Defines proper responses to imminent collision 
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https://bit.ly/2IX5eBo 


Safe Longitudinal Distance 
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= Proofs dont eliminate uncertainty 


e Need knowledge of environment & other 
vehicle equipment capabilities © 2021 Philip Koopman 54 


Omax,accel During 
Response Time p 





F=MA 
It's not just a 
good idea. 


It’s the Law! 
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Even though Newtonian Physics is useful 
e It requires accurate world model information (from perception??) 
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Validating an Autonomous Vehicle Pipeline even sity 
1 Y 
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TRAJECTORY VEHICLE > 
LU ma 
Y) : O 
Machine Randomized Control Autonomy <¢ 
Learning & Heuristic Systems Interface To 
Based Algorithms Vehicle 
Approaches = Control 
=> Run-Time Software > Traditional 
> Simulation Safety Envelopes Validation Software 
& SOTIF = Doer/Checker = Doer/Checker Validation 
approaches Architecture Architecture 


Prediction & perception are uniquely difficult to assure 
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Free space: available drivable area 


e Move to where the free space is going to be 
e Can require fine grain classification 






NOT wi 
HAS BEEN. 


Wayne Gretzky 
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https://www.azquotes.com/ quote/117311 





From Fail Silent to Fail Operational Nelo 
= Driver Assistance approach Sa ej 
e Driver controls vehicle >. — 
e Computers help 
e Fail silent computers 
= ADS approach VA 


e Computer controls vehicle > 
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e Driver is out of the loop during operation —=—=— #4328 Feb 2021 hitps://bit.yadParxz 
e Computers keep working after a failure (“fail operational”) 

— At least long enough for driver to take over in Level 3 

— More redundancy than conventional vehicle 

— Different fault management (e.g., pull to side of road) 
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FAIL-DEGRADED ELECTRICAL NETWORK 








MAIN ELECTRICAL NETWORK 
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Figure 24. Implemented Redundancy Concept in the BMW ADS 


BMW VSSA https://bit.ly/3gCiiGw 
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Redundancy & Decomposition (ee 
= ASIL B(D) redundancy strategy: 
e Two ASIL B channels for net ASIL D 
e Failure independence required! 





= Mitigate potential common cause failures: 
e Same perception/sensor fusion/planning algorithms 
e Same operating system, compiler, libraries, ... 
e Same CPU types, network chips, discrete components, ... 
e Same hardware boards (thermal; EMC; power distribution) 


= Attaining high diversity (>90%) is difficult! 
e Requires significant, dedicated engineering effort 
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Move To Centralized Architecture 


= Older architecture 
e ECU per major function 
e 1°' Tier supplier does HW + SW + integration for ECU 


= Newer architecture 
e Central computing ECU 


— Sensor fusion + path planning + vehicle control 
— Other functionality as well 


e Supplier + OEM software on same ECU 


= Multi-function and multi-vendor software integration 
e Resource & functionality conflict management by OEM 
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Changing 
Computing Architecture 








“> Feature specific ECUs =} centralization 
‘“» Fail silent > fail operational strategy 


“> Significant effort on redundancy+diversity 
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